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ABSTRACT 

This testimony reports on the state of program 
evaluation in the field of drug education for youth. The report 
focuses on the role of evaluation in: (1) the Department of 
Education's program of grants to states and local school districts 
under the Drug- Free Schools and Communities Act (DFSCA) of 1986, as 
amended; (2) the department's Drug-Free School Recognition Program; 
and (3) in other, promising, comprehensive community-based programs. 
Information in the report came from the Department of Education, 
interviews with officials and the examination of documents such as 
states 1 biennial DFSCA progress reports, application forms and 
reviewer resumes in the Recognition Program, and contractor 
evaluation reports. The report looks separately at the role of 
evaluation in each of the three areas listed above and notes that 
there is slow progress in establishing the results of sizable federal 
funding, or in learning what works in the field of drug abuse 
prevention for youth. It is concluded that the field of drug abuse 
prevention for youth faces two major challenges of evaluation method 
and capacity: determining the results of prevention programs and 
stimulating evaluation by state and local agencies.. (NB) 
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Mr* Chairman and Members of the Subcommittee: 

I am pleased to be here today to provide updated information 
on the state of program evaluation in the field of drug education 
for youth. Your invitation asked us to review developments since 
our earlier reports, specifically on the role of evaluation 

in the Department of Education's program of grants to 
states and local school districts under the Drug-Free 
Schools and Communities Act (DFSCA) of 1986, as 
amended, 

in the department's Drug-Free School Recognition 
Program, and 

in other, promising, comprehensive community-based 
programs • 

To prepare this testimony, we gathered new information from 
the Department of Education. We interviewed officials and 
reviewed documents such as states' biennial DFSCA progress 
reports, application forms and reviewer resumes in the 
Recognition Program, and contractor evaluation reports. 

In summary, we found slow progress in establishing the 
results of sizable federal funding, or more generally, in 
learning what works in the field of drug abuse prevention for 
youth. I will amplify this observation as I address each of the 
three areas you asked about in turn. 

EVALUATION OF SCHOOLS' ACTIVITIES UNDER DFSCA 

The 1989 amendments to the act required states to evaluate 
the effectiveness of local district programs and also required 
districts to report to the state on the method used to evaluate 
their own efforts and the results of such evaluation. 1 

Our Earlier Findings on Evaluation 

In a 1990 evaluation, we reviewed what school districts were 
doing under the act in 1988-89 and concluded that little was 
known at the local, state, or national level about what approach 
works best or how effectively the various programs and curricula 



2 Funds are distributed annually to the states via the state 
education agency (SEA) and the governor's office. Ninety percent 
of funds allotted to the SEA under the act are to be further 
distributed by a formula to local school districts (and also 
intermediate educational agencies or consortia of districts). 
The SEAs and governors' offices may use the funds for a variety 
of authorized activities. 
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reduce or prevent drug and alcohol use among students. 2 For 
example, the six big school districts we visited had not 
determined the effectiveness of their DFSCA programs. 

In our report, we quoted the predictions of department and 
school district officials that good evaluation was so difficult 
and costly that, in response to the strengthened evaluation 
requirements in the law, states would use readily available but 
weak indicators of student drug use, such as number of drug- 
related arrests, referrals, or school suspensions. Also, 
officials in the states and districts we visited planned to 
continue to report results in terms of numbers of participants or 
their opinions without assessing behavior. Our 1990 report noted 
that some gains in knowledge might be expected in the future from 
a number of evaluation-related activities sponsored by the 
department , including : 

contracting for a study of successful programs, 

contracting for a longitudinal study of the 
effectiveness of funded projects, and 

preparing a handbook to give guidance on evaluation to 
state and local drug education projects. 

Department Contractor Findings on Evaluation 

Also several years ago, a contractor examined evaluation at 
the state and local level, among other activities under DFSCA, 
from the passage of the act in 1986 through 1988-89. The 
contractor reported little activity, and what little there was 
centered on process and implementation, net results. The 
contractor also reported that staff in local programs at that 
time recognized the need for evaluation and wanted more direction 
about how to do it. The contractor recommended that the SEAs' 
and governors' programs use a substantial portion of their 
administrative funds under the act to strengthen monitoring, 
technical assistance, and evaluation, and that the department 
provide guidance to grantees on evaluation activities suited to 
different levels of resources. 3 

Activity and Progress Since the Earlier Reports 

Little progress has occurred in every respect. None of the 



2 Drug Education: School-Based Programs Seen as Useful but Impact 
Unknown (GAO/HRD-91-27, November 28, 1990). 

3 A Study of the Drug-Free Schools and Communities Act. Report on 
State and Local Programs: Executive Summary (Washington, D.C.: 
U.S. Department of Education, January 1992), pp. E-21, E-26. 



activities mentioned in our earlier report has produced any 
results as yet. The study of successful programs yielded no 
empirical findings the department could publish, and the 
effectiveness study is years away from reporting results (due in 
late 1995). The evaluation handbook has still not been issued. 
The department has not taken any significant action on either of 
the recommendations made by the contractor. 4 Regional centers 
authorized under the act to provide training to schools and 
others working on prevention have offered workshops and consulted 
with projects on evaluation , but the department could not 
identify any guidance making that topic a priority. 

We looked to see if anything had changed in state and local 
evaluation owing to the increased requirements in the 1989 
amendments. We first asked the department what they had learned 
from the evaluation data in the states' biennial reports. 
Officials said there were not enough staff with time to read 
them, and in any case, they had little incentive to consider the 
information, as they lack authority to deny funding based on any 
shortcomings of either reporting or performance. The department 
contracted with an outside firm to read the states' reports but 
will not get a summary or analysis until late this year, more 
than a year after the reports arrived in Washington. 

We therefore read some of the most recent reports ourselves, 
to see what states reported now that evaluation is required. We 
read reports submitted in 1992, covering 1989-91, from 10 states 



4 The department has generally given very little guidance on 
evaluation. The department has not issued regulations for DFSCA 
programs, but it has addressed how much evaluation is enough each 
year. Nonregulatory guidance issued in lieu of regulations on 
December 11, 1992, included a question about how a state should 
fulfill its annual evaluation responsibility. The department 
answered that states "must evaluate a significant number of 
programs each year" so that after 2 years the state "must have 
evaluated the effectiveness of all...." The department gave no 
further guidance as to what would be a satisfactory evaluation. 

Department officials gave as one reason for not offering more 
guidance to DFSCA programs about evaluation design or outcome 
measures that the statute uses only the single word 
"effectiveness" without elaboration. The statute does not, 
however, prohibit the department from suggesting alternatives for 
evaluating effectiveness. 

3 
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chosen at random, 5 



We found the same weaknesses in study design and data that 
observers saw in past years. Of the 10 states , only 3 reported 
that they evaluated local programs' effectiveness , and their 
methods were not strong. 6 In addition, three states said that 
local districts had evaluated program effectiveness, but methods 
again were weak in two of these three. The states' biennial 
reports of their own evaluations of projects under the governors' 
segment of DFSCA showed somewhat more activity (with five states 
reporting completing effectiveness studies in the period 1989- 
91). But we judged the methods here weak again; no report 
described effective programs based on sound data. All 10 of the 
states reported they had done statewide surveys of youth drug 
use. Some claimed they drew conclusions about the impact of 
programs from these survey findings of declining drug use rates — 
a design or approach we judge unsound chiefly because so many 
factors other than the programs can affect an entire state's 
aggregate youth drug use rates. 

Conclusions and Observations 

The scale of federal investment — $1.5 billion in the last 3 
years alone for the SEA and governors' programs — requires better 
progress in evaluation than we have observed. Seven of the 10 
state reports we reviewed listed no evaluation of local programs 
in the 2 -year period. Where we saw some evaluation, the methods 
used were weak, producing little reliable information. For its 



5 The 10 states we reviewed had collectively received $170.3 
million for SEA and local school district activity under DFSCA in 
fiscal years 1990 and 1991, the period covered by the reports we 
reviewed. This is about 23 percent of the total allocated to all 
states for these purposes (i.e. excluding the governors' funds). 
The largest state we examined had received $76.8 million in those 
2 years; the smallest, $3.7 million. 

6 The evaluation methods reported by these three states were: 
state agency staff observing during visits; interviewing and 
surveying of school administrators; collecting student opinions 
in focus groups; and tallying local reports of objectives 
achieved. 

4 
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part, the department has virtually ignored this shortfall, 7 

The fundamental barrier to more and better evaluation is the 
fact that funds are mostly allocated by formula — without any need 
to know what works. Arguably such a funding scheme, up to now, 
was needed to support building a broad base of capability 
(through purchase of materials, training, and so on). But now 
may be the time for a new approach that includes more deliberate 
targeting. 8 Thus, the Congress could split the allocation of 
funds. Part of the funds would flow, as before, by formula to 
states and districts (that would allow continuation of some level 
of activity everywhere) . The remainder would be awarded upon 
evidence of continuing need and of increasing targeting of 
resources on programs of proven effectiveness. The statute could 
direct that states award funds for local school districts in a 
parallel fashion, with a base amount for ongoing activity and a 
discretionary amount based on need, plans, and results. Use of 
evaluation data would thus be necessary within regular decisions 
at every level. The incentive is small for improving evaluation 
quantity or quality when it is not tied to the receipt of funds 
but is seen as serving only to meet a federal requirement. 

In addition, the Congress could change the statute to give 



department officials repeated to us their view (stated 
originally at the time of our 1990 report cited above) that local 
prevention program evaluation is extremely difficult. We 
continue to disagree. Evaluations need not involve sophisticated 
details such as random assignment of students or elaborate 
measurements. We think it is not beyond the resources of many 
schools, for example, to take careful measures of what services 
students received, together with assessment of whether the 
intended results were achieved (effects on knowledge, attitudes, 
or behaviors other than drug use) • Comparison is crucial as 
well, and can be done in many ways including repeated measures on 
those involved, as well as comparison to students not involved or 
past trends. The Department of Health and Human Services (HHS) 
has published a useful step-by-step guide showing the feasibility 
of local evaluation, Prevention Plus III: Assessing Alcohol and 
Other Drug Prevention Programs at the School and Community Level 
(Washington, D.C.: HHS, 1991). The guide was drawn from 
materials originally prepared by one of the Department of 
Education's regional centers. 

Responsible officials of the Department of Education provided 
comments on the preliminary findings and conclusions from the new 
data-gathering reported throughout this statement. We 
incorporated their comments where appropriate. The officials 
declined to comment on the alternative funding approach suggested 
below until the department's reauthorization proposal is 
submitted. 



priority to evaluation among the activities authorized for 
states' administrative funds from the act, and to add evaluation 
training and assistance as a priority among the tasks of the 
regional centers* 

The national evaluation already under way by the Department 
of Education is a good start, but much more could be done to 
strengthen evaluation even with no change in law. The fact that 
the law mentions only the one word "effectiveness" in describing 
the nature of the required evaluation allows the department to 
encourage a wide range of measures of effectiveness to be used. 
And targeted attention to a few key areas could pay off, also. 
Three states account for 25 percent of DFSCA state grant funds 
and seven states for 42 percent, so much could be learned by 
improved evaluation in those states alone. 

EVALUATION IN THE DRUG-FREE 
SCHOOL RECOGNITION PROGRAM 

The picture is brighter here, as the department has been 
more active in directing the evaluation approach used in giving 
out these national citations of merit and has taken a number of 
important steps to respond to our critique of past methods. 9 

How the Program Works 

The Recognition Program was established by the department in 
1987 to demonstrate, by example, the creation of safe, 
disciplined, and drug- free schools and to offer models from which 
others could learn. Public and private elementary and secondary 
schools are eligible, but must be nominated through a state 
education agency or any of several other specific sources 
designated by the department. Panels of teachers, parents, law 
enforcement officials, evaluation and research experts, and 
community organization members review applications, visit the 
top-ranked schools to check their claims in the application, and 
make final recommendations to the Secretary. The department 
awards no funds to the winners, only praise and publicity. 
(Recognized schools are invited to showcase their activities in 
regional and national meetings for the benefit of others, and the 
department puts out publications on the models.) 

Our Earlier Review of the Recognition Process 

After we examined in 1991 how the department evaluated 
applicants for recognition, we made a number of recommendations. 
One was to eliminate a policy that restricted the types of 
programs eligible, another was to widen the nomination process. 



9 Drug Abuse Prevention: Federal Efforts to Identify Exemplary 
P rograms Need Stronger Design (GAO/PEMD-91-15, August 22, 1991). 



We also recommended adding reviewers with research and evaluation 
qualifications, clarifying the criteria and data to be used to 
evaluate programs, and eliminating some questionable steps in the 
decision-making process. 

Most important, we found the department did not require that 
effectiveness be demonstrated, and we recommended that that be an 
unambiguous criterion. Under the past procedure, the public and 
educators interested in what works could not be assured that 
recognized programs did indeed cause valued outcomes. 

Progress Since Our Report 

The department has continued the program since our review of 
the 1989-90 cycle, completing two more cycles and awarding 126 
recognitions among 313 nominees; the 6th cycle is under way now, 
with 144 nominees from 32 states. We wanted to see if we could 
have any more confidence that — as a result of improved 
evaluation — this year's awards will truly signify programs that 
not only made a good effort but also had beneficial effects that 
would not have happened otherwise. 

In brief, the chances of that are better, and the results 
are promising. The department has made significant progress 
correcting the problems we identified in its evaluation 
approach. 10 First, the application package has been revised. A 
school that wants to be recognized is on notice that a review 
criterion will be "documented evidence that it is making progress 
in reducing' 1 drug use and incidents of violence and disruptive 
behavior. The applicant must address this criterion by answering 
a question about the "concrete evidence that your program is 
effective" and must discuss the "evaluation design, data 
collection instruments, and empirical indicators of success." 

Second, department officials told us they have added to each 
review panel an individual with training or experience in social 
science research or evaluation, as we recommended. We reviewed 
the resumes of the 10 individuals involved in the current cycle. 
Only 1 of the 10 lacked relevant capabilities (academic degrees 
or experience) for judging empirical evidence of effectiveness. 

Two of our staff independently reviewed the evaluation 
section of five applications in the current cycle without knowing 
the review panels' decisions whether or not to proceed with a 
site visit. These two reached complete agreement, which suggests 



10 Other areas have shown mixed progress on our recommendations. 
Nominations are more broadly and systematically sought and the 
separate Steering Committee that acted (without any additional 
evaluation information) as a second review panel, has been 
eliminated. The eligibility policy, however, is unchanged. 

7 
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that at least the process yields interpretable evidence. We 
agreed that three applicants had enough evaluation evidence to 
warrant the next step of review, while two did not. We then 
confirmed that our views aligned in all five cases with those of 
the panel. Of course, we did not have the resources to review a 
large enough group of applications to reach general conclusions 
about the current review process, but the trends are promising in 
the gathering and use of evaluation data. We have no additional 
suggestions for improved evaluation to make at this time. 

EVALUATION OF COMPREHENSIVE 
COMMUNITY-BASED PREVENTION PROGRAMS 

At your request, we studied these programs for early 
adolescents in 1990, using both a national survey of 138 programs 
reported to be exemplary and observations of 10 of them. 11 We 
hoped to find effective models of this type of program, since the 
approach is widely advocated. General features that distinguish 
this approach to prevention include addressing multiple 
dimensions of youths' lives (family, peer group, school, and 
community) and using a variety of services. The wide public 
interest in such programs is suggested by the demand for our 
report; it was one of the top 10 "best sellers" among all GAO's 
reports in 1992. 

Our Earlier Findings on These Programs 

As you recall, we called some of the programs "promising," 
since they appeared, on interim measures such as participants' 
involvement and program completion rates, to be doing well in 
achieving preconditions of long-term impact. This allowed us to 
note six features of those programs that deserved the attention 
of others designing drug abuse prevention efforts. But we found 
the programs rarely conducted evaluations that would be necessary 
to show whether or not they were effective in reducing or 
preventing alcohol and drug use. This lack of evidence, we 
believe, has hindered the programs' own development and 
refinement, slowed the overall development of the field, and — 
since the information gap persists — limited the information we 
have today to aid the Subcommittee in considering authorizations 
for different types of prevention efforts. 

About three quarters of the programs of this type that we 
surveyed received federal funds, so we noted a potential federal 
role in strengthening evaluation. We specifically recommended 
that the Department of Education finish the evaluation handbook 



x l Adolescent Drug Use Prevention t Common Features of Promising 
Community Programs (GAO/PEMD-92-2, January 16, 1992). 
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for DFSCA grantees discussed above. 12 

More general ly, our report raised for congressional 
consideration the reluctance of programs to spend scarce service 
funds on evaluation/ and we suggested that if the Congress wants 
to learn more about the effectiveness of such efforts/ special 
funds might be set aside for national evaluations. 

Further Progress in Learning 

What Works In This Type of Program 

Unfortunately/ here again we cannot report much progress in 
the growth of our understanding about the effectiveness of these 
programs. Some new information on community-based programs 
generally will come from a section of the department's^ DFSCA 
outcomes s 1y. The contractor is collecting data over time on 
the outcomes of 10 such programs funded under the governors' 
section of the act. Two of our staff independently examined the 
selected programs and agreed that three programs were somewhat 
similar to those we studied in that they were targeted to youth, 
comprehensive, and community- based. Results of this study will 
not be ready until 1995. 13 

We also reviewed the evaluation section of 10 state biennial 
progress reports on governors' programs to see if projects in the 
" high-risk youth" funding category had been the subject of any of 
the modest evaluation activity we reported above. As noted, only 
half the states said they had completed effectiveness evaluations 
of any activity under the governor's funding. Of those, one 
state reported contracting for a study of high-risk youth 
projects, but gave no results; the others reported no pertinent 
evaluations • 

Conclusion and Observation 

Again, the department has provided little guidance that 
would help these especially complicated programs do affordable 
evaluation, although there may be modest progress once that 



12 The handbook would help some of the kinds of programs we 
studied, as they can be DFSCA grantees under the governors' 
program where part of the funds must be used for "innovative 
community-based programs of coordinated services for high-risk 
youth." The Office of Substance Abuse Prevention, now called the 
Center for Substance Abuse Prevention, in the Department of 
Health and Human Services also sponsored such projects at the 
time of our review and had relevant evaluation materials under 
development. 

"The Robert Wood Johnson Foundation is supporting a national 
study of comprehensive prevention programs. 

9 
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guidance (the handbook) is produced* Only modest funding is 
potentially available, however, for community programs for high- 
risk youth in the governors' part of the act, which suggests this 
group of funded programs may never be the basis for learning a 
great deal about such programs. Thus, we conclude that for the 
Congress to know more about the effects of such programs, a 
national study will need to be requested, or even mandated, since 
it appears none will happen otherwise. Whatever department is 
tasked with the work (Education or HHS), other departments and 
agencies funding relevant projects should also be required to be 
involved. These programs cross many lines. 

CONCLUDING OBSERVATION 

In sum, we think this field clearly faces two major 
challenges of evaluation method and capacity: (1) determining the 
results of prevention programs, and (2) stimulating evaluation by 
state and local agencies. These two are always hard to do, and 
progress has been limited. One can still be optimistic, however, 
about effective prevention, even in the face of this lack of 
evidence. We have found promising efforts; others have 
successfully met the evaluative challenge; and with so much 
programmatic activity, it seems reasonable to believe that there 
are effective efforts in schools and communities to be measured, 
understood, and replicated. Progress in evaluation is essential, 
and it is certainly more difficult when a funding agency gives 
scant guidance. Yet major funding needs to rest on evidence of 
impact • 

Mr. Chairman, this concludes my statement. I will be happy 
to answer any questions you may have. 
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